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(54) Video-on-demand systems 

(57) A system and method for use in a distributed 
video-on-demand system of a type wherein at least one 
node (102,104,106,108) provides blocks (124) of video 
data to clients (111,112) and wherein at least some of 
the blocks (124) of video data are replicated on multiple 
nodes. Observed response characteristics for other 
nodes are recorded at least at a given one of the nodes 
which serves a client requesting a replicated block. The 



given one of the nodes also records response charac- 
teristics reported to it by the other nodes. Hie node from 
which to fetch the replicated data block is selected 
based on which nodes include a copy of the replicated 
data block and based on at least one of the observed 
response characteristics and the reported response 
characteristics. 
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Description 

The present invention relates to a video on demand 
(VOD) system of the type wherein multiple clients are 
serviced by video streams delivered from a central s 
video server. 

A distributed real-time client server environment 
includes multiple clients and servers where each client's 
file system makes real-time I/O requests (i.e., requests 
with deadlines) to the servers responsible for retrieving 10 
the requested data. An example of such a system is a 
video server system consisting of multiple front-end cli- 
ent nodes with network adapters and back-end nodes 
or storage servers. In the presence of replication, the 
same data block may be present on multiple servers. At is 
any random instant of time, the load (i.e.. the number of 
requests waiting to be served or in service) on different 
servers is likely to be different. Additionally, different 
servers may have different performance characteristics 
(e.g., transfer speed, seek time, etc.). 20 

A straightforward approach to server selection 
would be to randomly choose a server for an I/O opera- 
tion. This approach, however, may require the servers 
to be run at a lower utilisation to avoid missing dead- 
lines. 25 

According to the first aspect of the present inven- 
tion, there is provided in a distributed video server sys- 
tem of a type wherein at least one node provides video 
data to clients and wherein at least some of the data is 
replicated on multiple nodes, a method for selecting a 30 
node to serve the video data for provision to the clients, 
comprising the steps of: 

recording observed response characteristics, the 
observed response characteristics being response 35 
information concerning nodes in the video server 
system as observed by a given node; 
recording reported response characteristics, the 
reported response characteristics being response 
information concerning other nodes as reported to 40 
the given node by at least one other node in the dis- 
tributed video server system; and, 
selecting a node to serve replicated video data 
based on which nodes include a copy of the repli- 
cated video data and based on at least one of the 45 
observed response characteristics and the 
reported response characteristics. 

According to the second aspect of the present 
invention, there is provided a distributed video server so 
system, comprising: 

a plurality of nodes coupled by way of a first com- 
munication network; 

a plurality of clients coupled to a given one of the ss 
nodes; 

at least two of the nodes including a storage sub- 
system having replicated blocks of video data 
stored thereon; 
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means in the given one of the nodes, for recording 
observed response characteristics, the observed 
response characteristics being response informa- 
tion concerning nodes in the video server system 
as observed by the given one of the nodes; 

means, in the given one of the nodes, for recording 
reported response characteristics, the reported 
response characteristics being response informa- 
tion concerning other nodes as reported to the 
given one of the nodes by at least one of the other 
nodes; and 

means, in the given one of the nodes, for selecting 
a node to serve the replicated data block based on 
which of the nodes includes a copy of the replicated 
data block and based on at least one of the 
observed response characteristics and the 
reported response characteristics. 

According to the third aspect of the present inven- 
tion, there is provided in a video server system of a type 
wherein at least one node provides blocks of video data 
to clients and some of the blocks of video data are rep- 
licated on multiple storage devices, a method for select- 
ing a storage device from which to retrieve the video 
data for provision to the clients, comprising the steps of: 

recording observed response characteristics, the 
observed response characteristics being response 
information concerning performance of a plurality of 
the storage devices in the video server system as 
observed by the at least one node; 
recording reported response characteristics, the 
reported response characteristics being response 
information concerning the plurality of the storage 
devices as reported to the at least one node in the 
video server system by computer processes man- 
aging the storage devices; and 
selecting a storage device from which to retrieve 
the replicated data block based on which storage 
devices include a copy of the replicated data block 
and based on at least one of the observed 
response characteristics and the reported 
response characteristics. 

According to the fourth aspect of the present inven- 
tion, there is provided a distributed computing system, 
comprising: 

a plurality of nodes coupled by way of a first com- 
munication network; 

a plurality of clients coupled to a given one of the 
nodes; 

at least two of the nodes including a storage sub- 
system having replicated blocks of data stored ther- 
eon; 

a first table instantiated in a memory in the given 
one of the nodes, dedicated to recording observed 
response characteristics, the observed response 
characteristics being response information con- 
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cerning nodes in the system as observed by the 
given one of the nodes; and 
a second table instantiated in the memory, dedi- 
cated to recording reported response characteris- 
tics, the reported response characteristics being 
response information concerning other nodes as 
reported to the given one of the nodes by at least 
one other nodes. 

According to the fifth aspect of the present inven- 
tion, there is provided a method of controlling a distrib- 
uted computing system of a type wherein a plurality of 
nodes are coupled by way of a first communication net- 
work and wherein a plurality of clients are coupled to a 
given one of the nodes, at least two of the nodes having 
the capability of performing an identical function, com- 
prising the steps of: 

recording observed response characteristics, the 
observed response characteristics being response 
information concerning nodes in the system as 
observed by the given one of the nodes; 
recording reported response characteristics, the 
reported response characteristics being response 
information concerning other nodes as reported to 
the given one of the nodes by at least one other 
nodes; and 

selecting a node to perform the function based on 
which of the nodes can perform the function and 
based on at least one of the observed response 
characteristics and the reported response charac- 
teristics. 

The present invention includes a system and 
method wherein clients use information about the 
observed server response time, and server perform- 
ance measures to select a node to perform a given rep- 
licated function. The function can be, for example, 
provision of a data block available from more than one 
node. In a preferred embodiment, both the clients and 
the servers share server performance measures by pig- 
gybacking this information on normal messages passed 
between the clients and the servers as well as those 
passed between one server and another. 

The invention will now be described, by way of 
example, with reference to the accompanying drawings, 
in which: 

Fig. 1 is a block diagram of a distributed video-on- 
demand system according to an embodiment of the 
present invention; 

Figs. 2A-2C show the data structures used by the 
system of Fig. 1 ; 

Fig. 3A and 3B show the procedure used by the cli- 
ent to select a server; 

Fig. 4 shows the procedure used by the client to 
update the server performance measures; 
Fig. 5 shows the procedure used by the client to 
update the observed server response time; and, 
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Fig. 6 shows the format of a message including 
piggy backed performance data. 

The present system and method schedules reai- 

5 time I/O operation so as to improve VOD system 
throughput while satisfying real-time service require- 
ments. The approach takes into account the observed 
storage server response time (i.e the response time of 
the storage server that has the requested data), 

10 together with performance measures such as the 
expected storage server load and block placement 
among the storage servers, to select a block replica to 
be read. Each client has a corresponding file system 
(which can be shared with other clients). Whenever a 

is response is received from a storage server, the file sys- 
tems update a local copy of the observed storage 
server's response times and the time of day at which 
each response was observed. 

The expected load information on different storage 

20 servers is maintained in the following manner. Both the 
file systems and the storage servers maintain an array 
of load counts on each storage server and the associ- 
ated local timestamp at the storage server. The file sys- 
tems and the storage servers need not have 

25 synchronised clocks. This information is piggybacked 
on every message between the file systems and the 
storage servers. On receiving this information each 
node (file system or storage server) updates its load 
count array to reflect the latest load information for a 

30 storage server based on the associated timestamp. 
Using the above mechanism a file system obtains load 
information even about those storage servers it has not 
accessed recently. It should be understood that each 
node may contain a file system, a storage server or 

35 both. 

Fig. 1 is a block diagram of a distributed video-on- 
demand system according to an embodiment of the 
present invention. Computer systems (VOD System 
nodes) 102-108 are interconnected by way of a first 

40 (system) communication network (or switch) 110. The 
computer systems can be of the same or different types 
and can have different performance characteristics 
(from one another) depending on system type and con- 
figuration. One or more of the VOD nodes is also inter- 

45 connected with clients 111, 112 by way of a second 
(client) communication network 1 14. Each VOD System 
Node includes a system network interface 116 which 
provides an electronic and communication interface to 
the system communication network 110. Nodes con- 

so nected to the client communication network also include 
a client network interlace 120 which provides an elec- 
tronic and communication interface to the client commu- 
nication network 114. 

Each VOD System Node 102-108 includes a buffer 

55 memory 122 for the temporary storage of retrieved 
video blocks 124 and other messages and a processor 
(cpu) 126 which operates under control of various pro- 
grams residing in a main memory 128. At least some of 
the nodes 102-108 include disks 130 which have mov- 
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ies stored thereon. Each of the movies is made up of a 
number of the video blocks 124. The video blocks 124 
for a given video (e.g. movie) can be distributed (e.g. 
striped) across the disks 1 30 of a disk array within a sin- 
gle system or across the disks of multiple VOD system 5 
nodes. In the latter case, the video blocks are communi- 
cated from a VOD System Node that has a requested 
block on its disks to the VOD System Node serving a 
requesting client by way of the system communication 
network 110. The requested block is then sent to the 10 
requesting client by way of the client communication 
network 1 1 4. Attributes (size, space, play history, block 
mapping, etc.) of each video are stored in an attribute 
file 132 which is also stored on one or more of the disks 
130. Similar to the video-blocks, the attributes can also 15 
be communicated between VOD System Nodes by way 
of the system communication network 110. 

The programs in the main memory will now be 
described. It should be understood that each of the pro- 
grams causes the VOD System Node to perform a 20 
"process" under the program s direction. At least one of 
the computer systems has one or more client applica- 
tions 134 which communicate with the client's 111,112 
by way of the client communication network 114. The 
client applications also start, stop, pause and resume 25 
video playback upon client request. Similarly, at least 
one of the computer systems includes one or more file 
systems 136 which manage block placement and 
retrieval of the video blocks on the disks. The file sys- 
tems include an I/O Router 136a which is responsible 30 
for selecting the storage server from which to retrieve 
the requested block. 

Systems with disks also include a block server pro- 
gram 138 which is responsible for the physical retrieval 
of blocks from the disks and for providing the blocks to a 35 
requesting filesystem (which could be on any of the 
VOD System Nodes, including those without local 
disks). The block server 138 is also referred to herein as 
a "storage server" or simply a "server". 

The set of programs stored in the main memory 1 28 40 
also include control process modules 140 that, on vari- 
ous nodes, reserve a channel (i.e., resources) and set 
up viewing sessions before the start of video playback. 
Those of skill in the art will recognise that a number of 
other conventional software controlled processes, not 45 
described in detail here, are also involved in the control 
and support of the video server functions. 

The VOD System Nodes 102-108 can be embodied 
using any processor of sufficient performance for the 
number of video streams to be supported. For example, so 
a small capacity video server could be embodied using 
a RISC System/6000 TM system while a larger capacity 
server could be embodied using an ES/9000 TM system 
(both available form International Business Machines 
Corporation of Armonk, New York). The disks 130 can 55 
be embodied as any conventional disk subsystem or 
disk array. The communication networks 110, 114 can 
be, for example, fibre optic networks, conventional bi- 
directional cable networks or any other network of suffi- 
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ciently high bandwidth to handle the real-time require- 
ments of video data delivery in a VOD System. The 
clients can be embodied as set-top boxes 118, 120, or 
workstations or a combination of both. 

Fig. 2A shows the Server Load Table 200. The 
Server Load Table is a data structure maintained by 
both the block server 138 and the file systems 136 on 
each node for tracking load information about the block 
servers. Each VOD System Node maintains its own 
copy of the Server Load Table. Each copy is updated to 
reflect the latest loads by way of piggyback information 
sent along (from one node to another) with video data 
blocks. Specifically, a copy of the table is piggybacked 
by the file systems onto each request and by the block 
servers onto each response. 

Each entry in the Server Load Table 200 tracks 
(stores) the observed response time and load of each 
block server 138 in the VOD System Node. Each row 
contains the serverld 210, the observed delay 230 of 
the last response from that server, the delay timestamp 
220 containing the time at which the delay 230 was 
observed, and the server load 250 as reported by the 
server (e.g. server utilisation). The load timestamp 240 
is the server generated time stamp of the time at which 
the load 250 was reported. The load can be, for exam- 
ple, defined as the queue length (number of outstanding 
requests) at the server or as the server utilisation meas- 
ured as the number of requests served per unit time. 

The file system also maintains routing tables 136b. 
The Routing Tables include a Video Block Table 260 
(shown in more detail in Fig. 2B) and a Request Table 
285 (shown in more detail in Fig. 2C). The Video Block 
Table 260 contains a row for each video block. Each row 
contains the file identifier (fileld) 265 for the video block, 
the video block number (blockNo) 270, the number of 
replicas 275 of the video block and a list (Serv. Id) 280 
of the storage servers (and therefore the VOD System 
Nodes) that have the video block. In the example 
shown, file F1 block b81 has 1 replica on server S6 
whereas file F2 block b95 has three replicas on servers 
S3, S1, S10. 

The Request Table 285 is used to update the Delay 
fields 230 of the Server Load Table 200. Each row of the 
Request Table corresponds to an outstanding request 
and contains an identifier for the request (requestld) 
230. The Request Time field (requesffime) 295 is the 
time (according to the file system's clock) that the 
request was sent to the server and is set by the file sys- 
tem when the request is sent. 

The steps used by the file system to select a server 
when it is necessary to request a new file block are 
shown in Figs. 3A-3B. The server table 200 contains the 
observed response time and the reported server load. 
Both types of information are weighted as follows by 
appropriate confidence factors when deciding which 
server to select. In step 302, the file system searches 
the Video Block Table 260 using the fileld 265 and bloc- 
kNos 270 of the requested block to find the number of 
replicas 275 (denoted by n) and the server ids 280 
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(S1 Sn) of the servers on which these blocks reside. 

in step 304, the file system computes a delay confi- 
dence factor CFi.d for each server Si based on the cur- 
rent time t and the delay timestamp 220 (Ti,d). This 
factor measures the level of confidence of the file sys- 
tem that the observed delay 230 is still current. The 
delay confidence factor is low if Ti.d is in the remote 
past. In the present embodiment, the file system com- 
putes Cfi,d= l/(e ) where x is given by (t-Ti,d)/T and 
T is a pre-determined scaling factor. Similarly, in step 
306 the file system computes a load confidence factor 
Cfi.u for each server based upon the load timestamp 
240 as Cfi.u = 1/(e y ) where y is given by (t-Ti t u)/T . 

In step 308, the file system computes an overall 
delay confidence factor CFd as follows. It first computes 
Pd, the product of all the server delay confidence factors 
Cfi.d. Note that Pd will be high only if all the individual 
Cfi.d are high. Similarly, the file system computes Pu, 
the product of all the Cfi.u. Cfd is then computed as 
wd*Pd/(wd*Pd+wu*Pu) where wd and wu are weights 
that indicate the relative importance of the delay 240 
and the load 250. If both are to be given equal impor- 
tance, wd=wu=1 . In step 310 the load confidence fac- 
tor CFu is computed as wu*Pu/(wd*Pd+wu*Pu) . 

In step 312, the file system computes a delay bad- 
ness factor Bi,d for each server Si given by 
di/(d 1 +. . .+dn) . Bi.d is high if the delay of server i is high 
relative to the other servers. In step 314, the file system 
computes a load badness factor Bi,u for each Si given 
by ui/(u1+...un) . In step 316, the file system computes 
the overall badness factor Bi for each server Si as 
Cfi,d*Bi,d+Cfi,u*Bi,u . In step 318, the file system 
selects the server with the lowest badness factor Bi and 
exits in step 320. 

Fig. 4 shows the steps used by the file systems and 
servers to update the loads 250 in the Server Load 
Table 200. These steps are executed whenever the file 
system receives a message from the server or when the 
server receives a request from the file system. In step 
410, the server (or file system) extracts the update 
Server Load Table 610 (U) that was piggybacked on the 
request (or response). The format of the piggybacked 
message 600 including the update Server Load Table 
610 is shown in Fig. 6. The update Server Load Table 
610 is organised in the same way and includes the 
same fields as the Server Load Table 200. 

In subsequent steps, the server (or file system) 
scans the local copy of the Server Load Table 200 (S) 
and updates the appropriate load 250. In step 405, the 
index i is set to the index of the first row in the Server 
Load Table (S). In step 410, the load timestamp 240 
from table U 610 and the load timestamp 240 from table 
S 200 are compared. If the load timestamp 240 from 
table U is more recent, the server load 250 in table S is 
set to the value of server load (Load) 250 in table U. In 
step 420, a check is made to see if there are more serv- 
ers in table S. If so, the index i is set to the next row and 
step 410 is executed. If not, the update load procedure 
terminates in step 430. 
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Fig. 5 shows the steps used by the file systems to 
update the delays 230 in the server load table 200. This 
step is executed whenever a response is received from 
a server. In step 510, the file system locates the entries 

5 in the Request Table 285 corresponding to this request. 
In step 520, the response time for this request is com- 
puted as the difference between the current time and 
the request time 295. In step 530, the file system 
locates the row corresponding to this server in the 

w Server Load Table 200. The delay field 230 is set to the 
response time computed in step 520 and the delay 
timestamp field 220 is set to the current time. 

While the present embodiment is described in con- 
junction with load information being monitored at the 

is storage server level, the principles of the present inven- 
tion can also be readily applied at the disk level. Thus, 
observed and reported disk performance can be used, 
even in the context of a single node video server, to 
select a disk from which to obtain a video data block 

20 replica. 

Claims 

1. In a distributed video server system of a type 
25 wherein at least one node provides video data to 

clients and wherein at least some of the data is rep- 
licated on multiple nodes, a method for selecting a 
node to serve the video data for provision to the cli- 
ents, comprising the steps of: 

30 

recording observed response characteristics, 
the observed response characteristics being 
response information concerning nodes in the 
video server system as observed by a given 

55 node; 

recording reported response characteristics, 
the reported response characteristics being 
response information concerning other nodes 
as reported to the given node by at least one 

40 other node in the distributed video server sys- 

tem; and 

selecting a node to serve replicated video data 
based on which nodes include a copy of the 
replicated video data and based on at least one 
45 of the observed response characteristics and 

the reported response characteristics. 

2. A method according to Claim 1, wherein the 
reported response characteristics are sent to the 

so given node as information piggybacked on other 
messages passed between the nodes. 

3. A method according to Claim 1 or 2, wherein the 
observed characteristics include a measured delay 

55 in retrieving the video blocks. 

4. A method according to Claim 1 , 2 or 3, wherein the 
reported characteristics include a measure of per- 
formance of load on each node. 
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5. A method according to Claim 4, wherein the meas- 
ure of performance is reported along with a time 
stamp indicative of when the measure was taken. 

6. A method according to Claim 5, wherein the meas- s 
ure of performance is stored by the given node and 
wherein only the measure of performance having a 
most recent time stamp for each node is retained by 
the given node. 

10 

7. A method according to Claim 3, wherein the repli- 
cated block is retrieved from the node having the 
shortest measured delay. 

8. A method according to any one of Claims 1 to 6, is 
wherein the observed response characteristics and 
the reported response characteristics are each 
given a weight and wherein the selecting is also 
based on the weight. 

20 

9. A method according to Claim 8. wherein the select- 
ing is based on the response characteristics having 
the highest total weight. 

10. A method according to any one of the preceding 25 
Claims, comprising the further step of providing the 
replicated data block to at least one of the clients. 

11. A distributed video server system, comprising: 

30 

a plurality of nodes (102,104,106.108) coupled 
by way of a first communication network (110); 
a plurality of clients (111,112) coupled to a 
given one of the nodes (102,104,106,108); 
at least two of the nodes (102,104,106,108) 35 
including a storage subsystem (130) having 
replicated blocks (124) of video data stored 
thereon; 

means (136,138,200) in the given one of the 
nodes (102,104,106,108), for recording 40 
observed response characteristics, the 
observed response characteristics being 
response information concerning nodes in the 
video server system as observed by the given 
one of the nodes; 45 
means (136,138,200), in the given one of the 
nodes (102,104,106,108), for recording 
reported response characteristics, the reported 
response characteristics being response infor- 
mation concerning other nodes as reported to so 
the given one of the nodes by at least one of 
the other nodes; and 

means (1 36,1 38), in the given one of the nodes 
(102,104,106,108), for selecting a node to 
serve the replicated data block (1 24) based on ss 
which of the nodes includes a copy of the repli- 
cated data block and based on at least one of 
the observed response characteristics and the 
reported response characteristics. 
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12. A system according to Claim 11, wherein each of 
the nodes having a replicated copy of the block 
(124) includes means for sending the reported 
response characteristics to the given node as infor- 
mation piggybacked on other internode messages. 

13. A system according to Claim 11 or 12, wherein the 
observed characteristics include a measured delay 
(230) in retrieving the video blocks (124). 

14. A system according to Claim 12, wherein the 
reported characteristics include a measure of per- 
formance of load (250) on each node. 

15. A system according to Claim 14, wherein the 
means for sending include means for including a 
time stamp (240) indicative of when the measure of 
performance (250) was taken. 

16. A system according to Claim 15, wherein the given 
node includes means (200) for storing the measure 
of performance (250) and for retaining the measure 
of performance (250) having a most recent time 
stamp (240) for each node. 

17. A system according to Claim 13, wherein the 
means (136,138) for selecting retrieves the repli- 
cated block (1 24) from the node having the shortest 
measured delay (230). 

18. A system according to any one of Claims 1 1 to 16, 
wherein the observed response characteristics and 
the reported response characteristics are each 
given a weight and wherein the means (136,138) 
for selecting selects the node (102,104,106,108) at 
least in part based on the weight. 

19. A system according to Claim 18, wherein the 
means (136,138) for selecting, selects the node 
(102,104,106,108) having the highest total weight. 

20. In a video server system of a type wherein at least 
one node provides blocks of video data to clients 
and some of the blocks of video data are replicated 
on multiple storage devices, a method for selecting 
a storage device from which to retrieve the video 
data for provision to the clients, comprising the 
steps of: 

recording observed response characteristics, 
the observed response characteristics being 
response information concerning performance 
of a plurality of the storage devices in the video 
server system as observed by the at least one 
node; 

recording reported response characteristics, 
the reported response characteristics being 
response information concerning the plurality 
of the storage devices as reported to the at 
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least one node in the video server system by 
computer processes managing the storage 
devices; and, 

selecting a storage device from which to 
retrieve the replicated data block based on 
which storage devices include a copy of the 
replicated data block and based on at least one 
of the observed response characteristics and 
the reported response characteristics. 

21. A distributed computing system, comprising: 

a plurality of nodes (102,104,106,108) coupled 
by way of a first communication network (110); 
a plurality of clients (111,112) coupled to a 
given one of the nodes (102,104,106,108); 
at least two of the nodes (102,104,106,108) 
including a storage subsystem (130) having 
replicated blocks (124) of data stored thereon; 
a first table (200) instantiated in a memory in 
the given one of the nodes (102,104,106,108), 
dedicated to recording observed response 
characteristics, the observed response charac- 
teristics being response information concern- 
ing nodes in the system as observed by the 
given one of the nodes; and 
a second table (200) instantiated in the mem- 
ory, dedicated to recording reported response 
characteristics, the reported response charac- 
teristics being response information concern- 
ing other nodes as reported to the given one of 
the nodes by at least one other nodes. 
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as reported to the given one of the nodes by at 
least one other nodes; and, 
selecting a node to perform the function based 
on which of the nodes can perform the function 
and based on at least one of the observed 
response characteristics and the reported 
response characteristics. 



22. A system according to claim 21 further comprising: 

means (136,138) , in the given one of the 
nodes (102,104,106,108), for selecting a node 
to serve the replicated data block (124) based 
on which of the nodes includes a copy of the 
replicated data block and based on at least one 
of the observed response characteristics and 
the reported response characteristics. 

23. A method of controlling a distributed computing 
system of a type wherein a plurality of nodes are 
coupled by way of a first communication network 
and wherein a plurality of clients are coupled to a 
given one of the nodes, at least two of the nodes 
having the capability of performing an identical 
function, comprising the steps of: 

recording observed response characteristics, 
the observed response characteristics being 
response information concerning nodes in the 
system as observed by the given one of the 
nodes; 

recording reported response characteristics, 
the reported response characteristics being 
response information concerning other nodes 
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FIG. 2A 
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